Skip to content

Add instructions: research experiment discipline - #2854

Open
GuoCheng24 wants to merge 1 commit into
github:mainfrom
GuoCheng24:add-research-experiment-discipline
Open

Add instructions: research experiment discipline#2854
GuoCheng24 wants to merge 1 commit into
github:mainfrom
GuoCheng24:add-research-experiment-discipline

Conversation

@GuoCheng24

Copy link
Copy Markdown

Adds instructions/research-experiment-discipline.instructions.md — guidance for repositories where Copilot writes evaluation harnesses, benchmark sweeps, or any code that reports a performance number.

Falls under "Development methodologies and best practices" / "Testing strategies and quality assurance" in CONTRIBUTING. The gap it addresses: in research code the common failure is not a crash but a number that looks great and is wrong — models happily write evaluation code that tunes on the test set, compares against unmatched baselines, or reports point estimates inflated by group leakage. The instructions are specific and mechanical: null models scored first (freeze on a well-scoring null), calibration/held-out separation on the unit of independence, a held-out access budget, published-baseline reproduction before improvement claims, paired error bars, and every new check demonstrated to fail on a deliberately broken input.

Distilled from real failures in agent-run experiment campaigns; the longer-form methodology with runnable demonstrations lives at https://github.com/GuoCheng24/breakthrough-harness.

  • Branch created from main
  • npm start run; docs/README.instructions.md updated (the script also produced unrelated churn in docs/README.agents.md from live MCP-registry data — left out to keep the diff minimal; happy to include if the check requires it)

@github-actions github-actions Bot added instructions PR touches instructions new-submission PR adds at least one new contribution labels Aug 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔒 PR Risk Scan Results

Scanned 1 changed file(s).

Severity Count
🔴 High 0
🟠 Medium 0
ℹ️ Info 0

✅ No matching risk patterns were detected in changed files.

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

instructions PR touches instructions new-submission PR adds at least one new contribution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant